
United States Patent and Trademark Office 



UNITED STATES DEPARTMENT OF COMMERCE 
United States Patent and Trademark Oflice 
Address: COMMISSIONER FOR PATENTS 
P.O. Box 1450 

Alexandria. Virginia 22313-1450 
www.uspto.gov 



APPLICATION NO. 



FILING DATE 



FIRST NAMED INVENTOR 



ATTORNEY DOCKET NO. 



CONFIRMATION NO. 



10/796,222 



03/08/2004 



Jiun-Ming Wang 



20350 7590 09/29/2005 

TOWNSEND and TOWNSEND and crew, LLP 

TWO embarcadero center 

EIGHTH FLOOR 

SAN FRANCISCO, CA 94111-3834 



084260-OOOOOOUS 



4630 



EXAMINER 



XU» KEVIN K 



ART UNIT 



PAPER NUMBER 



2676 

DATE MAILED: 09/29/2005 



Please find below and/or attached an Oflice communication concerning this application or proceeding. 



PTO-90C (Rev. 10/03) 



Office Action SumniBrv 


Application No. 
10/796,222 


Applicant(s) 

WANG ET AL. 


Examiner 
Kevin K. Xu 


Art Unit 
2676 





» The MAILING DATE of this communication appears on the cover sheet with the correspondence address 
Period for Reply 



A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) OR THIRTY (30) DAYS. 

WHICHEVER IS LONGER, FROM THE MAILING DATE OF THIS COMMUNICATION. 

- Extensions of time may be available under the provisions of 37 CFR 1.136(a). In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 

- If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS from the mailing date of this communication. 

- Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S.C. § 133). 
Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce any 
earned patent term adjustment. See 37 CFR 1.704(b). 

Status 

1)S Responsive to communication(s) filed on 19 September 2003 . 
2a)[J This action is FINAL 2b)IEI This action is non-final. 

3) n Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quayle, 1935 CD. 11, 453 O.G. 213. 

Disposition of Claims 

4) JXI Claim(s) /V7 is/are pending in the application. 

4a) Of the above claim(s) is/are withdrawn from consideration. 

5) n Claim(s) is/are allowed. 

6) K1 Claim{s) 1-17 is/are rejected. 

7) D Claim(s) is/are objected to. 

8) n Clalm(s) are subject to restriction and/or election requirement. 

Application Papers 

9) n The specification is objected to by the Examiner. 

10) S The drawing(s) filed on 19 September 2003 is/are: a)n accepted or b)Kl objected to by the Examiner. 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1 .85(a). 
Replacement drawing sheet(s) including the correction is required if the drawing(s) is objected to. See 37 CFR 1.121(d). 

11) 0 The oath or declaration is objected to by the Examiner. Note the attached Office Action or form PTO-152. 

Priority under 35 U.S.C. § 119 

12) n Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 1 19(a)-(d) or (f). 
a)n All b)n Some * c)D None of: 

1. D Certified copies of the priority documents have been received. 

2. n Certified copies of the priority documents have been received in Application No. . 

3. n Copies of the certified copies of the priority documents have been received in this National Stage 

application from the International Bureau (PCT Rule 17.2(a)). 
* See the attached detailed Office action for a list of the certified copies not received. 



Attachment(s) 

1) M Notice of References Cited (PTO-892) 

2) □ Notice of Draftsperson's Patent Drawing Review (PTO-948) 

3) IS Information Disclosure Statement(s) (PTO-1449 or PTO/SB/08) 

Paper No(s)/Mall Date 9/19/2003 . 



4) □ Interview Summary (PTO-413) 

Paper No(s)/Mail Date. . 

5) □ Notice of Informal Patent Application (PTO-152) 

6) □ Other: . 



U.S. Patent and Trademark Office 
PTOL-326 (Rev. 7-05) 



Office Action Summary 



Part of Paper No./IVIail Date 10796222 



Application/Control Number: 10/796.222 
Art Unit: 2676 



Page 2 



Detailed Action 

Drawings 

The informal drawings are not of sufficient quality to permit examination. 
The drawings have "blacked out" regions and thus indistinct areas as observed in 
Figures 1a, lb, 3a, 3b, 3c, 3d, 5, 6, 7, 8, 9, 17a, 17b and 17c. 

Accordingly, replacement drawing sheets in compliance with 37 CFR 1.121(d) 
are required in reply to this Office action. The replacement sheet(s) should be labeled 
"Replacement Sheet" in the page header (as per 37 CFR 1.84(c)) so as not to obstruct 
any portion of the drawing figures. If the changes are not accepted by the examiner, the 
applicant will be notified and informed of any required corrective action in the next Office 
action. 

Furthermore, it has been noted that the applicant checked off a petition pursuant 
to 37 CFR 1 .84(a)(2) to accept color drawings. However, this application does not 
contain the actual petition pursuant document. Therefore applicant must resend the 
petition. 

Color photographs and color drawings are not accepted unless a petition filed 
under 37 CFR 1.84(a)(2) is granted. Any such petition must be accompanied by the 
appropriate fee set forth in 37 CFR 1 .17(h), three sets of color drawings or color 
photographs, as appropriate, and, unless already present, an amendment to include the 
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following language as the first paragraph of the brief description of the drawings section 
of the specification: 

The patent or application file contains at least one drawing executed in color. Copies of this patent or 
patent application publication with color drawing(s) will be provided by the Office upon request and 
payment of the necessary fee. 

Color photographs will be accepted if the conditions for accepting color drawings 
and black and white photographs have been satisfied. See 37 CFR 1.84(b)(2). 

Claim Objection 

Applicant is advised that should claim 4 be found allowable, claim 7 will be 
objected to under 37 CFR 1 .75 as being a substantial duplicate thereof. When two 
claims in an application are duplicates or else are so close in content that they both 
cover the same thing, despite a slight difference in wording, it is proper after allowing 
one claim to object to the other as being a substantial duplicate of the allowed claim. 
See MPEP § 706.03(k). Therefore one of the claims must be amended or cancelled. 

Claim Rejections - 35 USC § 102 

The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that 
form the basis for the rejections under this section made in this Office action: 
A person shall be entitled to a patent unless - 

(b) the invention was patented or described in a printed publication in this or a foreign country or in public 
use or on sale in this country, more than one year prior to the date of application for patent in the United 
states. 
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Claims 1-3 are rejected under 35 U.S.C. 102(b) as being anticipated by 
Matsumoto (6,356.272). 

Matsumoto teaches the making of a colorful three dimensional model comprising 
of inputting three-dimensional original measured data by explaining image data to be 
obtained by shooting the object of interest (Fig. 3, item 100) from a plurality of 
viewpoints with camera (Fig 3, item 120) carried by an operator moving around 
stationary object (Fig. 3, item 100). By identifying the position of camera ( Fig 3, item 
120) and the shooting direction for each shot of an image three dimensional shape 
model can be reconstructed (Fig 6D, item 300). This is a method of capturing image 
data to be inputted for reconstruction. (Col 16, lines 49-62) 

Matsumoto additionally teaches the making of a colorful three-dimensional model 
comprising of reconstructing mesh models with regular data by presenting FIG. 3 as a 
schematic block diagram showing a structure of a three-dimensional model generation 
apparatus (Fig. 3) to reconstruct a three-dimensional model from an actual object 
according to a first embodiment of the present invention. Referring to FIG. 3, an object 
of interest 100 is mounted on a turntable 110. Turntable 1 10 has its angle of rotation 
controlled according to, for example, a control signal from a computer 130. A camera 
120 shoots the rotating object of interest 100 at every specified angle. The 
obtained image data is applied to computer 130. Data of the shooting condition 
such as the rotary pitch of turntable 110 and the like is applied to computer 
130 from an input device 140. (Col. 15, lines 33-43) 
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Matsumoto also teaches making a colorful three-dimensional model comprising 
of abstracting color information by showing the case of picking up a total of n images 
by rotating an object of interest for every predetermined angle of rotation, as shown in 
FIG. 1 . In this case, each image information corresponds to the label number of 1 , 2, 3 
. . . , n. The object of interest is represented as a shape model (wire frame 
model) 300 using a polygon (triangular patch). When texture information is to 
be assigned to shape model 300, color information (texture information) of the 
image information of a corresponding label number is assigned for each 
triangular patch according to the direction of the camera shooting the object 
of interest. (Col 2, lines 1 1-21) 

Matsumoto further teaches making a colorful three-dimensional model * 
comprising of harmonizing color of texture images by showing the applying of color 
information (texture information) obtained from reference image information that has 
the greatest amount of texture information is assigned to each three-dimensional 
shape constituent element (polygon 27, figure 6D) forming shape model 300. (figure, 
6D) (Col 19, line 21 - Col 22, line 5) 

Matsumoto further teaches making a colorful three-dimensional model 
comprising of pixel blending to overlapped texture images between the mesh models. 
As to a purpose for blending, Matsumoto teaches the central projection system is 
disadvantageous in that the joint of the texture is noticeable when the gloss or the 
texture of the color information is slightly different due to the illumination and the like 
since the texture information is assigned from different. image information (image 
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information of a different label number) to a three-dimensional shape constituent 
element that is not present within the same range of rotation angle when viewed from 
the axis of rotation. (Col 2, line 44-52); said means for assigning texture information to 
a three-dimensional shape constituent element by carrying out a weighted mean 
process according to the area of a three-dimensional shape constituent element 
projected on each object image information on the basis of object image information 
corresponding to the related label number and the object image information 
corresponding to a predetermined number of label numbers including that related label 
number. (Col 7, lines 24-32); said according to a still further aspect of the present 
invention, a texture inforrnation assignment apparatus for a shape model includes: 
means for capturing a plurality of object images information by shooting an object of 
interest from different viewpoints; means for describing the shape of the object of 
interest as a shape model by a set of a plurality of three-dimensional shape constituent 
elements; and means for assigning texture information obtained by carrying out a 
weighted mean process for all the object image information according to the area 
corresponding to the three-dimensional shape constituent element projected on the 
plurality of object images information for every three-dimensional shape constituent 
element. Preferably, the means for assigning texture information to the three- 
dimensional shape constituent element obtains the area projected on the object image 
information for each three-dimensional shape constituent element, and uses the 
obtained area as the weighting factor in carrying out the weighted mean process. For 
the texture information of the three-dimensional shape constituent element, the portion 
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of the three-dimensional shape constituent element projected on the object image 
information is obtained. The image information (color, density or luminance) of this 
projected portion is subjected to a weighted means process to result in the texture 
information. (Col 8, lines 24- 46). Therefore claim 1 is anticipated by Matsumoto. 

Matsumoto moreover teaches the method claimed in claim 2, wherein the mesh 
model reconstructing step comprises of selecting a generic model according to 
measured data by showing computer 130 (Fig. 4) extracts a silhouette image from the 
image information corresponding to each shooting angle according to the image 
information applied from camera 120 (Fig. 4) to generate a three-dimensional shape 
model (Col 15, lines 45-48); said adjusting dimension and spatial position of the 
generic model to overlap with the original measured data by showing according to yet 
an additional aspect of the present invention, a three-dimensional model generation 
apparatus of generating a three-dimensional model of an object of interest includes: 
silhouette generation means for generating a plurality of silhouette images of the object 
of interest, estimation means for estimating the existing region of the object of 
interest in a space according to the plurality of silhouette images; and means for 
generating a three-dimensional model of the object of interest using the object of 
interest existing region obtained by the estimation means (Col 12, lines 41-50) 
Further, the space defined by voxels (3-D space according to the absence/presence of 
a three-D lattice point.) is referred to as voxel space 251. (Fig 7) Voxel space 251 is 
arranged with a size and position that encloses the object to be recognized. Here, 
this voxel space 251 is represented with the cylindrical coordinate system that can 
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represent the shape of a target object in a more natural manner with respect to the 
pickup of an image while rotating object 100 (Fig 6A) of interest. (Col 17, lines 38-47) 
; said mapping data of generic model with original measured data to deform the 
generic model data to be close to the original measured data by showing at step SIS 
(Fig 5), the texture corresponding to each polygon 27 (Fig 11) generated at step SI 6 
(Fig 5) is obtained from the object image to be mapped on each polygon 27. (Col 40, 
lines 22-25). Therefore claim 2 is anticipated by Matsumoto. 

Matsumoto in addition teaches the method as claimed in claim 3, wherein the 
color abstracting step is to establish texture-mapping relationship between two 
dimensional image of the original measure data and the generic model, which 
comprises seeking mapping points of mesh points of the generic model on the original 
measured data and triangles having the mapping points by showing at step SI 6 
(texture mapping, Fig 5), a plurality of three-dimensional shape constituent elements 
(for example, a polygon such as a triangular patch; for the sake of simplification, the 
three-dimensional shape constituent element is represented as a polygon hereinafter) 
27 (Fig. 1 1) on the basis of the three-dimensional shape of target object 100 obtained 
at step S14. The three-dimensional shape of target object 100 obtained at step S14 is 
represented by a plurality of polygons 27 (Col 40, lines 14-19); said calculating 
corresponding texture coordinates of the mapping points by showing the amount of 
texture information is determined according to the degree of match between the normal 
vector of each three-dimensional shape constituent element (polygon 27) and the 
normal vector of the image shooting plane parallel to the direction in which the 
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reference image was shot. IVIore specifically, the reference image that is most 
positively opposite the relevant polygon 27 is selected as the reference image having 
the greatest texture information with respect to that polygon 27 (Col 20, lines 26-34) ; 
said checking continuity of the triangles on the texture images by the weighted mean 
process of texture information from a predetermined number of reference image 
information, texture information for a corresponding polygon can be obtained. 
Therefore, texture information improved in texture continuity can be assigned to the 
relevant polygon. (Col 28 lines 49-55) Therefore claim 3 is anticipated by Matsumoto. 

Claim Rejections - 35 USC § 103 
The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 

obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

Claims 4-1 1 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Matsumoto. 

Matsumoto teaches all of the claimed limitations of claim 4 except for the 
rearranging sequence of measured data according to the overlapped relationship. 
However, Matsumoto teaches the joint of the texture is noticeable when the gloss or 
the texture of the color information is slightly different due to the illumination and the 
like since the texture information is assigned from different image information (image 
information of a different label number) to a three-dimensional shape constituent 
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element that is not present within the same range of rotation angle when viewed from 
the axis of rotation. (Col 2, lines 44-52). Therefore, it would have been obvious to one 
of ordinary skill in the art at the time the present invention was made to employ the 
teachings of Matsumoto which shows the joint of the texture is noticeable when texture 
of color information is slightly different, and thus, implies calculating color adjustment 
(Ai) of the texture image of each original measured data as a means for color 
harmonizing of the colorful 3-D model. Also referring to claim 4, Matsumoto teaches 
the image information (color, density or luminance) is subjected to a weighted mean 
process to result in texture information. (Col 7, lines 45-47). Thus, it would have been 
obvious to one of ordinary skill in the art at the time the present invention was made to 
employ the teachings of Matsumoto to utilize a weighted mean process as a method of 
adjusting color average of overlapped area. Lastly, again referring to claim 4, 
Matsumoto teaches the representation of data consisting of n three dimensional mesh 
models by teaching a three-dimensional shape model can be represented by a set of, 
for example, polygons (triangular patches). The aforementioned image information 
implies numeric information representing the luminance, color, or the gray level 
corresponding to each pixel output from camera 120. However, representation of a 
three-dimensional model is not limited to such a representation method. For example, 
a three-dimensional model can be represented as a group of surface shape elements 
of different shapes. Therefore, the shape that is the element for representing a shape 
model is generically referred to as a three-dimensional shape constituent element. • 
(Col 15, lines 48-59). Although Matsumoto does not explicitly state the rearranging 
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sequence of measured data according to overlapped relationship and magnitude of 
overlapping area, it would have been obvious to one of ordinary skill in the art at the 
time the present invention was made to rearrange a sequence of data in a certain way 
in order to fit the purpose of color harmonizing within overlapped area. Therefore claim 
4 would have been obvious. 

Claims 5-7 are similar in scope to claim 4 and thus are rejected under similar 
rationale. 

Consider claim 8, Matsumoto teaches the image information (color, density or 
luminance) of this a projected portion is subjected to a weighted mean process to result 
in the texture information. (Col 7, lines 45-47) Further, Matsumoto teaches the means 
for assigning texture information to the three-dimensional shape constituent element 
obtains the area projected on the object image information for each three-dimensional 
shape constituent element, and uses the obtained area as the weighting factor in 
carrying out the weighted mean process. For the texture information of the three- 
dimensional shape constituent element, the portion of the three-dimensional shape 
constituent element projected on the object image information is obtained. The image 
information (color, density or luminance) of this projected portion is subjected to a 
weighted means process to result in the texture information, (Col 8, lines 36-47), 
Although an equation for color average, A = (Ai.i * Wi,i + ....Ai,Ai.i*Wi,WM) /(Wi.i +...Wi,i. 
i) is not explicitly defined by Matsumoto, it would have been obvious to one of ordinary 
skill in the art at the time the present invention was made to utilize the calculation as 
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claimed in Matsumoto because if color information is obtained and subjected to a 
weighted mean process (as taught by Matsumoto) then that same technique would be 
a way of calculating color adjustment using mesh influenced weighted value in a 
straightforward manner with minimum calculations and reduced processing time. 

Claims 9-1 1 are similar in scope to claim 4 and thus are rejected under similar 
rationale. 

Claims 12-17 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Matsumoto (6,356,272) in view of Migdal et al (6,356.263 hereinafter "Migdal") 

In regard to claim 12, Matsumoto teaches a means for assigning texture 
information to a three-dimensional shape constituent element by carrying out a 
weighted mean process according to the area of a three-dimensional shape constituent 
element projected on each object image information on the basis of object image 
information corresponding to the related label number and the object image information 
corresponding to a predetermined number of label numbers including that related label 
number. (Col 7, lines 24-32) Although Matsumoto does not explicitly suggest seeking 
overlapped images covered by each triangle within overlapped areas for pixel blending, 
it would have been obvious to one of ordinary skill in the art at the time the present 
invention was made to utilize the texture processing of Matsumoto for the pixel 
blending as claimed because texture processing requires seeking of overlapped 
images covered by each triangle within overlapped areas. 
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Matsumoto further teaches the image information (color, density or luminance) of 
this a projected portion is subjected to a weighted mean process to result in the texture 
information. (Col 7, lines 45-47) Further, Matsumoto teaches the means for assigning 
texture information to the three-dimensional shape constituent element obtains the 
area projected on the object image information for each three-dimensional shape 
constituent element, and uses the obtained area as the weighting factor in carrying out 
the weighted mean process. For the texture information of the three-dimensional 
shape constituent element, the portion of the three-dimensional shape constituent 
element projected on the object image information is obtained. The image information 
(color, density or luminance) of this projected portion is subjected to a weighted means 
process to result in the texture information. (Col 8, lines 36-47) Although Matsumoto 
does not explicitly teach calculating pixel weight average to mapping area 
corresponding to each triangle, it would have been obvious to one of ordinary skill in 
the art at the time the present invention was made to utilize the method as claimed in 
Matsumoto because using an amount of image area obtained as a weighting factor in 
the averaging process would also be applied for calculating pixel weight average to 
mapping area. 

Lastly Matsumoto fails to explicitly teach calculating distances of vertices of each 
of the triangles within the overlapped areas to nearest edges of corresponding mesh. 
However, this is what Migdal (6,356,263) teaches. Migdal teaches non-symmetric 
criteria included using a distance function that would calculate a distance from one of 
the vertices. (Col 7, lines 30-34). Furthermore, Migdal teaches a procedure that 
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determines whether the square of the distance between the vertices' coordinates of the 
edge to be subdivided is greater than a threshold value. (Col 7, lines 55-58). It would 
have been obvious to one of ordinary skill in the art at the time the present invention 
was made to combine the teachings of distance calculation between a vertex and an 
edge of a triangle taught by Migdal into a pixel blending technique using weighted 
average taught by Matsumoto because the use of the distance function taught by 
Migdal provides computational and processing efficiencies such as saving of memory 
space and processor resources. (Col 8, lines 8-10) and by calculating these distances 
a more accurate 3-D model would be realized. 

Claims 13-17 are similar in scope to claim 4 and thus are rejected based on 
Matsumoto as explained in the previous rejection under 35 U.S.C. 103(a) 

Conclusion 



Any inquiry concerning this communication or earlier communications from examiner 
should be directed to Kevin K Xu whose telephone number is 571-272-7747. The 
examiner can normally be reached on Monday-Friday from 8:30 AM - 5:00 PM. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Matthew Bella can be reached on (571) 272-7778. 
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Information regarding the status of an application may be obtained from Patent 
Application Information Retrieval (PAIR) system. Status information for published 
applications may be obtained from either Private PAIR or Public PAIR. Status 
information for unpublished applications is available through Private PAIR only. For 
more infonnation about the PAIR system, see http://Dair-direct.uspto.qov . Should you 
have questions on access to the Private PAIR system, contact the Electronic Business 
Center (EB) at 866-217-9197 (toll-free). 
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